Spatio - Temporal Modeling of Dynamic 3 D Scenes from Visual Data

This thesis addresses the problem of modeling the time-varying shapeof a dynamic scene, as seen from a sparse multi-camera system. No restrictions areplaced on the scene either on the type and number of the actors in the scene, or onthe nature of their interactions. A priori, computer vision algorithms of surface re-construction based on silhouettes and multi-view stereo are performed individuallyat various frames, to obtain a sequence of meshes in 3D. These meshes are not con-sistent with each other, either geometrically or topologically.This thesis makes twocontributions towards obtaining a coherent spatio-temporal model of the underly-ing scene : (1) obtaining a temporally coherent segmentation of the mesh sequence,identifying parts that are moving in an approximately rigid manner (2) estimatingdense 3D motion on the surface through a mesh evolution framework, that handlestopological and geometrical inconsistencies amidst individual mesh reconstructions.In order to make these contributions, various sub-problems are solved. A newmethod of mesh segmentation is presented that is suited for visually reconstructedmeshes, which is robust to the geometric artifacts that occur in such reconstruc-tions. Segments extracted at various frames are then refined and repositioned into aconsistent segmentation of the mesh sequence. As a by-product of this scheme, ar-ticulated motion is estimated on the scene, as a set of rigid body transformations ofeach segment over the sequence. This pair of motion estimates and temporally con-sistent shape, constitute the underlying spatio-temporal model of the scene, whichis captured at a coarse level by the segmentation scheme. The latter chapters ofthe thesis explain how to estimate motion densely, to capture true non-rigid defor-mation of surfaces. In order to achieve this, sparse features are detected on thesurfaces and matched across time, in a geometrically valid manner. These sparsematches provide an initialization for estimating motion densely over the surface. Amesh evolution framework maps meshes reconstructed at two frames completely, ac-counting for the topological and geometrical inconsistencies, providing a means fortransferring motion estimates from one surface to the other. Dense 3D trajectoriesof surface points are thereby estimated over long sequences.The proposed methods are systematically tested on diverse and challenging datasets,which are made publicly available on the web. Qualitative and quantitative experi-mental results are presented in each chapter to validate the approach adopted.